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INTRODUCTION 


The  development  of  cost  estimating  tools  virtually  came  to  a  standstill 
after  the  surge  of  RAND  and  other  studies  ended  in  the  early  1970s.  The  cost 
estimating  "tool  box"  used  to  estimate  today's  systems  is  based  on  methodologies, 
problems,  and  technologies  that  are  pushing  20  years  of  age.  Some  of  the 
tools  are  not  adequate  to  estimate  today's  systems  with  their  new  technology. 

The  Air  Force  Systems  Command  (AFSC)  Cost  and  Management  Analysis  Directorate 
has  developed  a  comprehensive  long  range  program  to  restock  the  estimating 
"tool  box”  with  new  estimating  capabilities. 

'The  program  is  called  the  Cost  Methods  Improvement  Program  (CMIP)  and  was 
formed  with  the  development  of  the  AFSC  Cost  Analysis  Improvement  Group  (CAIG) 
in  1981.  The  CAIG  is  chaired  by  the  Director  of  Cost  and  Management  Analysis 
at  HQ  AFSC  and  has  as  members  each  of  the  AFSC  Product  Division's  Cost  Analysis 
Directors.  The  CAIG  prioritizes  and  approves  funding  for  research  projects 
which  benefit  more  than  one  Product  Division.  The  CMIP  is  managed  at  HQ  AFSC 
with  most  of  the  individual  projects  passed  on  to  the  Product  Divisions. 

The  AFSC  CAIG  was  formed  to  mirror  a  similar  group  formed  at  HQ  Air  Force. 

The  purpose  of  the  CAIG,  is  as  its  name  implies,  to  improve  the  cost  analysis 
function  within  AFSC.  At  the  onset  the  CAIG  started  a  research  program  aimed 
at  cost  estimating.  The  reason  a  concerted  effort  was  made  in  this  area  was 
that  little  cost  estimating  research  was  being  accomplished  at  that  time. 

The  first  year  of  funding,  1982,  was  a  minimal  effort  of  only  $200,000.  Since 
that  time  the  funding  and  number  of  projects  has  increased.  Funding  is  now 
supported  in  the  Air  Force  Program  Objective  Memorandum  through  Fiscal  Year 
1991  at  a  total  of  $8M. 

The  first  few  years  of  the  CMIP  saw  various  projects  started  and  completed. 
These  included  software  data  base  generation,  host  vehicle/subsystem  integration 
costs  and  Very  High  Speed  Integrated  Circuit  (VHSIC)  cost  relationships. 

Three  projects  were  of  a  long  term  nature  and  are;  the  Cost  Analysis  Handbook, 
Production  Rate  Model  and  Microelectronic  Road  Map.  These  projects  were 
conceived  to  be  dynamic  projects  which  would  change  as  technology,  cost 
relationships,  and  other  factors  changed. 

One  of  the  biggest  and  highly  supported  projects  was  the  development  of 
the  Cost  Analysis  Handbook.  "This  handbook  is  designed  to  be  used  like  an 
engineers  handbook.  Every  cost  analyst  in  the  Air  Force  will  get  one  for 
their  desk.  The  handbook  is  primarily  for  the  new  cost  analyst  and  is  to 
provide  them  with  tips,  shortcuts,  and  rules  of  thumb  that  an  analyst  would 
learn  after  years  of  experience.  The  handbook  is  to  help  speed  up  the 
learning  process,  as  well  as  provide  a  good  detailed  reference  for  the 
experienced  analyst. 
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PROJECT  SELECTION 


Research  projects  are  submitted  by  the  Product  Divisions  to  HQ  AFSC  for 
funding  consideration.  These  projects  are  then  discussed  among  the  members 
of  the  CAIG  to  determine  final  prioritization  and  funding.  The  projects 
under  consideration  represent  the  Product  Division's  needs  for  the  upcoming 
year.  These  needs  show  where  new  estimating  "tools"  are  required. 

HQ  AFSC  had  no  indication,  other  than  the  list  of  projects,  of  what  the 
long  term  needs  of  the  product  divisions  were.  With  the  development  of  the 
CMIP  Road  Map,  a  means  for  assessing  the  long  term  needs  was  realized.  Because 
the  Road  Map  was  developed  with  the  help  of  all  the  product  divisions,  the  HQ 
AFSC  CAIG  now  has  more  than  a  list  of  projects  to  help  them  prioritize  the 
funding  and  planning  of  projects. 

The  Road  Map  helps  the  prioritization  process  by  dividing  areas  of  need 
into:  (1)  New  Research  Areas,  (2)  Technology  Areas,  (3)  Other  Research  Areas, 

and  (4)  Survey  Areas.  These  four  categories  emerged  during  the  Road  Map 
study.  A  cost  estimating  framework  was  first  developed  in  which  the  changing 
nature  of  the  cost  estimating  problem  could  be  classified  with  respect  to 
advancing  technology  and  to  the  development  life  cycle.  Research  Areas  requiring 
improved  cost  estimating  methods  were  identified  within  this  framework.  Interviews 
were  then  conducted  with  key  Air  Force  Product  Division  and  Tecolote  cost 
estimators.  Modifications  to  the  Research  Area  list  were  made  and  specific 
topics  in  Technology  Areas,  Other  Research  Areas,  and  Survey  Areas  were  also 
identified. 

The  research  areas  include: 

-  Quantify  State-of-the-Ar t 

-  Extrapolation  Estimators  for  Evolutionary  Advance 

-  Technology  Breakthrough  Evaluation  Prior  to  DSARC  II 

-  DSARC  I  Cost  Estimating  Methods 

-  Technology  Impacts  on  Acquisition  Costs 

-  Technology  Impact  on  State-of-the-Art  System  Characteristics 

-  Normal  FSD  Schedule/Cost  Estimate 

FSD  Cost  Consequence  of  Technology  (Schedule)  Compression 
Methods  to  Estimate  Advance  Development  Costs 

These  research  areas  represent  longstanding  problems  that  cause  great 
heartburn.  Getting  to  the  bottom  of  some  of  them  may  be  extremely  difficult, 
if  not  impossible,  as  many  are  institutional  in  nature.  Institutional  problems 
represent  problems  with  the  bureaucracy,  "the  way  things  are  done."  For 
example,  trying  to  lock  in  a  system  definition  prior  to  DSARC  I  is  impossible 
and  will  always  cause  system  cost  estimating  problems.  Integration  cost 
estimates  are  also  required  very  early  in  a  subsystem's  life  to  determine  if 
it  should  be  installed  on  certain  platforms.  At  the  time  tht.se  estimates  are 
required,  the  subsystem  is  far  from  a  final  definition,  yet  "yes,"  "no" 
decisions  are  expected.  These  research  areas  must  be  studied  to  see  if  there 
is  a  better  way  to  estimate  costs  even  with  the  uncertainty. 


The  Technology  Areas  were  a  fallout  of  the  interview  process  mentioned 
above.  Technology  identification  was  not  the  intent  of  the  Road  Map.  However, 
it  was  apparent  in  the  discussions  that  the  following  technologies  were  of 
high  interest: 

-  Very  High  Speed  Integrated  Circuits  (VHSIC) 

-  Computer  Aided  Design  (CAD) 

-  Computer  Aided  Manufacturing  (CAM) 

-  Gallium  Arsenide  (GaAs) 

-  Composite  Materials 

These  technologies  appear  to  be  affecting  our  estimating  capabilities 
soonest.  These  technologies  will  impact  our  Cost  Estimating  Relationships 
(CERs)  and  may  change  the  attributes  (physical  or  performance)  that  are  cost 
drivers  in  the  CERs. 

The  other  Research  Areas  represent  areas  which  are  more  specific  than  the 
Research  Areas  identified  earlier.  These  areas  include: 

Software  Size  Estimating 
Historical  Schedule  Inefficiencies 

-  Estimate  Quality 

-  Non-Primary  Mission  Equipment  Cost 

-  Host  Vehicle/Subsystem  Integration  Costs 

-  Commonality  Problems 

-  Software  Cost  Estimating 

-  Evaluation  of  Estimates  Made  by  Price 

In  addition,  some  areas  identified  have  had  significant  research  previously 
completed  and  a  survey  of  the  area,  disseminating  the  ideas  is  the  next  logical 
step.  THese  have  been  identified  as  Survey  Areas  and  include: 

-  Correcting  for  Biased  Cost  Drivers 
Imbedded  Cost  Estimating  Practices 

-  Competition  Impacts 

-  Multiyear  and  Warranties 
Production  Rate  Effects 

-  Contractor/Subcontractor  Integration  Costs 

Although  some  of  the  areas  identified  above  include  past  CMIP  projects, 
the  Road  Map  brings  together  the  views  of  all  product  divisions  on  a  particular 
project.  The  benefit  is  that  when  one  product  division  may  feel  a  certain 
project  is  "hot"  and  benefit  other  product  divisions,  the  Road  Map  brings  the 
Command's  needs  into  perspective  and  that  project  can  be  evaluated  from  a  top 
level  viewpoint.  It  may  turn  out  that  the  other  product  divisions  may  not 
need  the  project.  By  having  existing  projects  in  the  Road  Map,  a  long  term 
need  is  established  for  these  project  areas  and  consequently  continued  funding 
and  high  prioritization  are  needed. 


EXCERPTS  FROM  THE  ROAD  MAP 


Excerpts  from  the  Research  Road  Map  study  follow.  They  have  been  extracted 
from  "Air  Force  Systems  Command  Cost  Research  Road  Map,"  by  C.  A.  Graver,  CR- 
0076,  Tecolote  Research,  Inc.,  July  1985.  This  document  will  be  available 
from  AFSC/ACC  on  or  about  2  October  1985. 

Highlighted  in  this  section  will  be  all  of  Section  1  and  2  and  parts  of 
Sections  4  through  8.  The  excerpts  will  then  be  followed  by  a  brief  summary. 
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INTRODUCTION 


STUDY  OBJECTIVE 

Headquarters,  Air  Force  Systems  Command  (AFSC)/ACC  is  charged  with  the 
responsibility  of  providing  timely  and  accurate  cost  estimates  for  future 
USAF  acquisitions.  Cost  estimating  techniques  used  to  accomplish  this  mission 
have  been  developed  over  the  last  twenty-five  years.  These  techniques  are 
based  on  solving  estimating  problems  associated  with  previous  and  current 
technologies.  As  such  they  are  not  always  appropriate  for  estimating  the 
cost  of  acquisitions  that  incorporate  new  technology  or  use  new  contracting 
methods. 

Over  the  past  ten  years,  cost  research  has  largely  been  focused  on  applying 
these  previous  cost  estimating  methods  and  techniques  to  current  estimating 
problems.  Little  attention  has  been  given  to  the  examination  of  the  entire 
estimating  process.  Are  there  fundamental  estimating  problems  which  have  not 
been  addressed?  Is  there  a  need  for  entirely  different  estimating 
technologies  which  require  new  estimating  approaches? 

The  purpose  of  this  study  is  to  step  back  and  take  this  overview  of  the 
cost  estimating  problem  and  then  to  define  new  cost  research  directions. 

Specific  research  tasks  are  not  to  be  delineated.  Also,  the  study's  focus  is 
on  estimating  capabilities  and  techniques  as  opposed  to  institutional  problems 
such  as  staff  training  or  the  automation  of  estimates. 

The  secondary  objective  is  to  identify  research  problems  of  common  interest, 
across  the  Product  Divisions.  These  topics  are  candidates  for  AFSC  funding, 
whether  they  are  new  research  areas  or  extensions  of  existing  research  or 
techniques. 

Specific  cost  estimating  problems  without  broad  interest  are  not  candidates 
for  the  cost  research  roadmap  unless  they  can  be  used  as  examples  to  develop 
and  demonstrate  a  new  technique. 

Also  excluded  from  the  study  are  the  following: 

Operations  and  Support  (O&S)  cost 

Institutional  and  procedural  changes 

Automation  intended  to  increase  estimating  efficiency 

Technical  recruiting 

Black  world  problems 
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STUDY  APPROACH 


Tecolote  Research,  Inc.  (Tecolote)  was  asked  to  conduct  this  study. 
Initially  three  steps  were  envisioned,  as  paraphrased  from  the  statement 
of  work.* 

1.  Compile  a  List  of  Cost  Research  Areas 

Tecolote  will  compile  a  list  of  cost  research  areas,  primarily 
through  brainstorming  sessions  with  senior  Product  Division 
and  Tecolote  personnel.  Formal  study  plans  will  be  reviewed 
as  a  background  to  the  interviews.  However,  the  primary  source 
of  information  will  be  the  interview  process  in  that  the  primary 
focus  of  the  study  is  on  study  areas  not  being  addressed  and 
on  requirements  for  new  techniques.  Current  study  plans  will 
be  a  good  starting  point  for  the  discussion. 

Tecolote  will  contact  as  many  Product  Divisions  as  feasible. 
Interviews  will  also  be  made  with  non-Air  Force  personnel  as 
resources  permit. 

2.  Organize  Ideas  into  Research  Areas 

Items  surfaced  during  the  interview  process  will  be  organized 
into  research  areas.  These  research  areas  may  have  common 
technical  thrust,  data  requirement,  generic  problem,  or  other 
unifying  dimension. 

3.  Establish  Priorities 

Priorities  will  then  be  established  within  and  across  the  research 
areas.  The  criteria  will  consist  of  timing  (when  will  it  be 
needed?) ,  importance  (how  significant  is  the  cost?)  and  doability 
(promising  techniques,  data  sets?) .  The  result  will  then  be 
an  aggregated  "game  plan"  for  cost  research. 

STUDY  CONDUCT 


In  reality,  after  the  first  interviews,  it  became  apparent  that  a  framework 
was  required  in  which  the  various  dimensions  of  the  cost  estimating  problem 
(procurement  phase,  technology  requirements,  etc.)  could  be  synthesized 
and  holes  could  be  identified  where  existing  cost  estimating  techniques 
were  insufficient.  Such  a  framework  was  developed,  based  on  the  author's 
experience  and  remarks  from  the  initial  interviews.  New  research  areas, 
where  new  techniques  need  to  be  developed,  were  defined  from  this  framework. 


‘Extracted  from  study  overview  sent  to  the  Product  Divisions,  prior  to 
interviews. 


Subsequent  interviews  were  used  to  expand  and  more  precisely  define 
this  framework.  In  addition,  these  interviews  brought  up  additional  specific 
estimating  problems. 

These  ideas  were  cross-referenced  into  research  areas  which  fell  into 
three  broad  groups: 

1.  new  research  areas  (identified  from  the  framework) 

2.  new  technologies 

3.  other  research  areas. 

An  evaluation  was  made  of  each  identified  research  area.  This  included 
the  following: 

Definition  -  a  short,  precise  definition  of  the  area. 

Significance  -  a  qualitative  statement  about  the  significance  of  the 
research  area. 

Product  Division  interest  (Applicability)  -  a  simple  check  for  Product 
Division  expressing  interest  (from  the  interviews)  or  having  possible 
interest  due  to  the  potential  applications. 

Discussion  -  a  free  flowing  discussion  of  the  research  area. 

Related  Areas  -  Identification  of  how  this  area  relates  to  other  research 
areas. 

Doability  -  an  assessment  of  the  difficulty  to  achieve  significant 
success  in  the  research  effort. 

Priority  -  Significance  and  doability  are  combined  into  a  rating  of 
extreme,  high,  medium,  or  low. 

Recommendation  -  a  recommendation  as  to  whether  or  when  the  research 
topic  should  be  pursued,  based  on  priority. 

Research  Steps  -  an  outline  of  research  steps  for  research  areas 
recommended  with  extreme  or  high  priority. 

During  the  research  area  presentations,  remarks  from  interviewees  are 
often  cited.  The  Product  Division  will  also  be  identified  so  that  common¬ 
ality  of  interest  may  be  assessed.  Product  Division  personnel  are  generally 
from  the  ACCs.  In  the  case  of  non-government  personnel,  the  Product  Division 
most  closely  supported  by  the  individual  will  be  cited.  This,  of  course, 
does  not  imply  any  direct  employment  affiliation.  Furthermore,  Product 
Division  sanction  is  not  given  to  any  of  the  remarks  from  government  or 
non-governmental  personnel. 
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REPORT  ORGANIZATION 


The  remainder  of  this  report  is  contained  in  seven  sections.  The  Framework 
is  contained  in  Section  2.  New  research  areas  identified  in  Section  2  are 
described  in  Section  3,  following  the  format  discussed  above.  Interaction 
between  the  New  Research  Areas  is  shown  in  Section  4.  Technology  topics  are 
contained  in  Section  5.  They  are  candidates  for  development  of  cost  research 
methods  defined  in  section  3. 

Other  research  areas  are  contained  in  Section  6.  Significant  work  has 
already  been  done  in  many  of  the  areas.  In  some  cases,  it  appears  that  a 
critical  survey  of  techniques  would  serve  the  AFSC  cost  community  better  than 
another  research  effort.  In  such  cases,  the  topic  is  discussed  in  Section  7, 
Survey  Topics. 

Final  recommendations  are  then  summarized  in  Section  8,  This  includes  a 
priority  sorting  across  the  research  areas. 

Cross  references  between  topics  is  made  throughout  the  report.  A  shorthand 
notation  was  adopted  to  facilitate  the  cross  referencing.  These  are  listed 
below  along  with  the  topics: 

1.  New  Research  Areas  (described  in  Section  3)  are  referenced  with  Roman 
numerals  ( I — IX ) . 

I.  Quantify  State-of-the-Art  (SOA) 

II.  Extrapolation  Estimators  for  Evolutionary  Advance 

III.  Technology  Breakthrough  Evaluation  Prior  to  DSARC  II 
(Uses  V  and  VI) 

IV.  DSARC  I  Cost  Estimating  Method (s) 

V.  Technology  Impact  on  Acquisition  Cost 

VI.  Technology  Impact  on  SOA  System  Characteristics 

VII.  Normal  FSD  Schedule/Cost  Estimate  (Schedule  Normalization) 

VIII.  FSD  Cost  Consequences  of  Technology  (Schedule)  Compression 

IX.  Methods  to  Estimate  Advanced  Development  Costs 

2.  Technology  Areas  (Section  5)  are  referenced  by  a  number  following  the 
letter  T  (T1-T5) . 

Tl.  Very  High-Speed  Integrated  Circuits  (VHSIC) 

T2.  Computer-Aided  Design  (CAD) 

T3.  Computer-Aided  Manufacturing  (CAM) 

T4 .  Gallium  Arsenide  (CaAs) 

T5.  Composite  Materials 

3.  Other  Research  Areas  (Section  6)  are  referenced  by  number  following 
the  letter  0  (01-08) . 

01.  Software  Size  Estimating 

02.  Historical  Schedule  Inefficiencies 

03.  Estimate  Quality 

04.  Non-Primary  Mission  Equipment  Costs 

05.  Host  Vehicle/Subsystem  Integration  Costs 

06.  Commonality  Problems 

07.  Software  Cost  Estimating 

08.  Evaluation  of  Estimates  Made  by  PRICE 
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4.  Survey  Areas  (Section  7)  are  referenced  by  number  following  the 
letter  S  (S1-S6)  . 

51.  Correcting  for  Biased  Cost  Drivers 

52.  Imbedded  Cost  Estimating  Practices 

53.  Competition  Impacts 

54.  Other  Contracting  Method  Problems 

55.  Production  Rate  Effects 

56.  Contractor  Integration  Cost  Estimation 
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FRAMEWORK 


The  biggest  problem  in  defining  future  research  directions  that  have  not 
been  sufficiently  addressed  by  the  cost  community  is  to  impose  seme  structure 
on  the  overall  cost  analysis  problem.  Within  this  structure  areas  can  be 
identified  where  the  cost  analysis  community  has  done  a  good  job  and  others 
where  improvement  is  needed.  The  following  four  charts  show  the  structure. 

They  show  the  cost  problem  as  it  relates  to  system  requirements  growth,  life- 
cycle  phase,  technology  breakthroughs,  and  schedule/cost  tradeoffs  in  full 
scale  development  (FSD) .  Roman  numerals  in  circles  on  these  charts  refer  to 
research  areas  which  appear  not  to  have  been  (sufficiently)  addressed  by  the 
cost  community.  These  research  areas  are  discussed  further  in  section  3. 

CHART  1:  TECHNOLOGY  AND  COST  FROM  A  SYSTEM  ORIENTATION 

When  making  a  cost  estimate  of  a  proposed  system,  the  analyst  has  to  make 
sure  that  the  cost  estimating  tools  have  been  developed  for  the  estimating 
problem.  These  estimating  problems  will  differ  with  the  degree  of  technology 
advance  required  in  the  system  as  a  whole  or  in  specific  subsystems. 

Technology  advance  can  be  classified  into  three  different  levels.  First, 
State-of-the-Ar t  (SOA)  requirements  apply  to  subsystems  which  have  been  built 
or  are  so  close  to  subsystems  which  have  been  previously  built  that  no  advance 
in  technology  is  required.  Here,  our  traditional  cost  estimating  tools  are 
extremely  good.  Analogies  will  usually  exist,  and  CERs  built  for  interpolation, 
such  as  those  produced  by  least-squares  regression,  are  often  available. 

The  main  estimating  concerns  are:  (1)  various  management  questions  that 
affect  the  cost  or,  (2)  specific  cost  components  where  existing  estimating 
techniques  can  be  improved.  Management  questions  include  all  the  new  contractual 
clauses  affecting  production  costs  such  as  warranties,  the  impacts  of  competition, 
etc.  Support  equipment  is  an  example  of  a  cost  component  that  could  use  CER 
improvement.  These  important  considerations  need  study,  but  they  basically 
represent  fine-tuning  an  already  good  estimate. 

The  second  technology  level,  called  evolution,  refers  to  extensions  of 
existing  technology  which  will  normally  happen  given  enough  time.  They  do 
not  depend  on  any  great  technical  breakthrough.  Traditional  estimating  techniques 
have  not  explicitly  been  built  for  this  type  of  estimating  problem.  They 
have  not  made  use  of  the  available  data  to  test  the  power  of  the  estimating 
techniques  for  such  extrapolations. 

There  have  been  some  attempts  to  account  for  technology  growth  by  a  time 
factor,  but  these  really  don't  score  the  estimating  procedure  on  how  well  it 
extrapolates.  What  is  needed  are  estimating  techniques  that  have  demonstrated 
successful  extrapolation  in  the  required  direction  of  technology  growth. 

This  will  be  new  research  area  II.  For  the  present,  techniques  which  have 
demonstrated  extrapolation  capabilities  will  be  called  extrapolation  estimators. 


CHART  1 


TECHNOLOGY  AND  COST: 
SYSTEM  ORIENTATION 


TECHNOLOGY 

REQUIREMENTS 

1  COST 

1  DESCRIPTION 

METHODS 

AVAILABILITY 

SOA 

1 

1  INTERPOLATION 

1  CERs 

1  ANALOGIES 

1 

GOOD; 

MANAGEMENT  QUESTIONS 
EXIST 

EVOLUTION 

1 

1  ROBUST 

1  ESTIMATORS 

NEGLIGIBLE  (T} 

DONE  TOO 
LATE 


BREAKTHROUGH 


SPECIAL 

STUDIES 


Little  formal  work  has  been  done  in  this  area,  although  procedures  are 
available  for  testing  estimating  techniques  for  extrapolation.  Some  existing 
estimating  techniques  (CERs)  will  hold  up  well  when  tested  for  extrapolation 
others  will  not. 

The  third  level  of  technology  growth  is  one  that  requires  a  technical 
breakthrough.  What  is  desperately  needed  is  an  acceptable  technique  for 
examining  the  system  physical/performance  characteristics  to  quickly  identify 
when  a  breakthrough  is  required  (this  constitutes  research  area  I  as  shown  in 
chart  1) .  Such  a  technique  will  also  be  able  to  examine  system  (subsystem) 
requirements  and  classify  technology  growth  into  any  of  the  three  categories 
discussed  above.  (Candidates  for  this  technique  exist;  they  need  to  be  examined 
and  one  selected.) 

When  a  breakthrough  requirement  is  identified,  the  role  of  the  cost  analyst 
is  first  to  find  out  what  breakthrough  the  Program  Office  (PO)  is  depending 
on.  If  the  answer  is  none,  then  a  big  red  flag  should  be  raised.  The  system 
is  in  trouble  and  the  cost  estimate  should  reflect  the  risk,  when  a  specific 
breakthrough  is  identified  by  the  PO,  special  cost  studies  are  required. 

These  special  studies  will  constitute  research  area  III.  In  general,  these 
studies  are  started  far  too  late  (or  not  at  all) .  The  time  to  begin  is  in 
the  laboratory  or  during  concept  validation.  Refer  to  chart  3. 

CHART  2:  TIMING  OF  ESTIMATES  WITHIN  THE  ACQUISITION  LIFE  CYCLE 

DOD  level  reviews  are  held  at  key  milestones  in  the  system  development. 

These  reviews  are  made  by  the  Defense  System  Acquisition  Review  Committeee 
(DSARC) .  As  originally  envisioned,  three  reviews  were  to  be  held  which  had 
to  be  successfully  completed  prior  to  the  system's  passing  into  the  next 
phase  of  the  acquisition  life  cycle,  as  shown  below: 


Phases 

Concept  Formulation 
Concept  Validation 
Full  Scale  Development 
Product  ion 


Phases 


DSARC  I 


DSARC  II 


DSARC  III 


During  Concept  Formulation,  alternative  combinations  of  technology  are 
examined  to  find  a  single  combination  that  will  the  threat.  Approximate 
levels  of  system  performance  are  determined  and  the  concept  presented  at 
DSARC  I.  The  work  normally  consists  of  paper  studies  administered  by  future 
sysystems  analysis  groups  (XR)  within  the  product  division. 


During  Concept  Validation,  the  concept  is  tested  to  show  that  high-risk 
technologies  can  be  made  to  work  together.  Engineering  prototypes  sufficient 
to  validate  the  concept  are  developed  and  tested.  Specific  physical 
characteristics  of  the  new  technologies  are  selected.  Work  is  often  performed 
in  government  labs  or  by  outside  contractors,  administered  by  a  fledgling 
Program  Office  (PO).  Resolution  of  all  technical  risk  areas  is  presented  at 
DSARC  II.  Tasks  performed  between  DSARC  I  and  II  are  often  referred  to  as 
Advanced  Development. 

During  Full  Scale  Development  (FSD) ,  or  Full  Scale  Engineering  Development 
(FSED) ,  prototypes  of  the  entire  system  are  developed  and  tested  to  demonstrate 
system  performance  and  producibility.  Work  is  most  often  performed  by  a 
defense  hardware  (software)  contractor  under  the  guidance  of  the  System  PO. 

Results  are  presented  at  DSARC  III,  where  a  production  decision  is  made. 

This  ideal  description  of  the  process  will  be  used  in  discussing  the 
research  areas  presented  in  this  study.  It  is  recognized  that  this  ideal  is 
not  always  followed. * 

Chart  2  addresses  the  changing  nature  of  the  cost  problem  as  the  system 
moves  through  the  DSARC  review  process.  It  is  connected  with  the  first  chart 
in  that  cost  problems  prior  to  and  including  DSARC  I  and  II  tend  to  address 
evolutionary  or  breakthrough  advances  in  technology,  at  least  for  some  subsystems. 

Prior  to  DSARC  I,  the  system  description  is  vague,  there  are  many  technical 
problems,  and  hence  the  subsystem  performance  levels  necessary  to  meet  DOD 
requirements  are  only  vaguely  defined.  Yet,  estimates  are  required  for  FSD 
and  production,  and  these  numbers  will  be  attached  to  the  system  concept  and 
be  used  in  system  evaluation  by  Congress.  How  are  reasonable  (unbiased) 
estimates  made  with  such  uncertainties?  Traditional  techniques  do  not  address 
this  problem.  This  will  constitute  research  area  IV. 

A  related  problem  at  DSARC  I  is  estimating  the  cost  of  concept  validation, 
the  advanced  development  phase.  Techniques  and  data  for  these  costs  are 
almost  nonexistent.  This  will  constitute  research  area  IX. 

Between  DSARC  I  and  II,  cost/performance  tradeoff  curves  based  on  the  new 
technologies  need  to  be  developed  so  that  specific  physical  and  performance 
characteristics  can  be  selected.  Less  detailed  curves  should  be  developed 
prior  to  DSARC  I  for  technology  selection.  Development  of  these  curves  is 
research  area  III. 


*The  hand-off  of  responsibility  from  XR  to  the  PO  does  not  always  happen 
at  DSARC  I.  Furthermore,  more  than  one  set  of  technologies  may  be  carried 
into  Concept  Validation,  especially  in  high-risk  programs  with  tight  deadlines. 
Similarly,  not  all  technical  problems  have  been  resolved  prior  to  FSED. 
Surthermore,  a  low-rate  initial  production  (LRIP)  may  be  started,  after 
an  initial  DSARC  III  review  and  prior  to  the  final  DSARC  III  review. 
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Roman  numerals  in  circles  (e*g.,  (v)  )  refer  to  the 
New  Research  Areas  described  in  section  3- 


By  DSARC  II  the  technology  problems  have  presumably  been  resolved  and  the 
system  defined.  The  cost/performance  tradeoff  curves  should  form  the  basis 
of  the  estimate  in  the  new  technology  areas.  Hopefully  they  will  also  have 
been  used  in  the  system  definition  process.  Estimating  problems  shift  to 
estimating  cost/ schedule  tradeoffs  in  FSD  (research  area  VII)  and  cost  impacts 
of  new  production  technology  (cost  research  area  V) . 

By  DSARC  III  the  system  definition  is  very  detailed  and  the  technology 
problems,  other  than  those  of  high-rate  production,  should  have  been  resolved. 
Prototype  costs  are  known  and  production  estimates  using  existing  techniques 
should  have  high  quality.  Only  the  contractual  form  and  management  questions 
should  remain  a  cost  estimating  problem. 

CHART  3:  TECHNOLOGY  BREAKTHROUGH  COST  PROBLEMS 

What  then  are  the  cost  problems  when  the  system  characteristics  require  a 
technology  breakthrough?  (Identification  of  the  breakthrough  is  assumed,  or 
the  system  is  in  trouble.)  Breakthrough  cost  impacts  must  be  estimated, 
research  area  V. 

The  first  estimating  problem  is  "How  will  production  costs  change?"  If 
the  breakthrough  is  or  requires  a  new  production  technique,  industrial 
engineering  (IE)  studies  are  required  to  see  how  the  new  technique  would  be 
applied  to  previously  built  systems  and  to  thereby  estimate  how  the  production 
costs  would  change.  However,  this  will  only  establish  a  single  point  on  a 
cost/performance  curve,  so  extrapolation  from  these  studies  is  an  unresolved 
cost  problem. 

The  next  problem  is  "How  will  PSD  costs  change?"  The  big  unknowns  are 
design  and  testing  factors.  Fabrication  costs  should  relate  to  production 
estimates. 

Another  problem  is  "How  will  the  SOA  change  with  the  new  technology?" 

The  new  technology  may  make  previously  far-out  system  advances  into  SOA 
applications.  The  Technology  classification  tool  called  for  in  Chart  1  may 
be  the  technique  to  describe  the  change  in  the  SOA.  This  will  be  research 
area  VI. 

Finally,  "How  will  the  technology  be  applied?"  This  is  related  to  the 
point  above.  DOD  rarely  uses  a  breakthrough  only  to  reduce  costs.  More 
generally,  DOD  buys  more  capability,  and  this  may  even  be  an  evolution  from 
the  new  SOA  called  for  in  the  previous  paragraph.  This  consideration  is  part 
of  research  area  IV. 
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New  Research  Areas  described  in  section  3* 
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CHART  4;  SCHEDULE/COST/TECHNOLOGY  INTEGRATION 

A  big  problem  in  FSD,  which  is  only  accentuated  when  a  technology 
breakthrough  is  required,  is  the  establishment  of  an  FSD  program  with  cost, 
schedule,  and  technology  advance  requirements  all  balanced.  The  central 
question  is:  how  does  one  make  a  "normal"  FSD  schedule,  assuming  that  FSD 
cost  is  to  be  minimized  and  that  no  accelerated  technology  growth  requirements 
are  present?  This  question  constitutes  research  area  VII. 

To  accomplish  this,  schedule  estimating  relationships  for  normal  FSD 
programs  will  have  to  be  developed  from  a  (nonexistent)  schedule  data  base 
which  has  been  normalized  for  elongated  programs  (stretch-outs  due  to  budget 
cuts)  or  accelerated  programs  (affecting  schedule  only  or  schedule  and 
technology) .  (Note  that  these  two  accelerated-porgram  effects  are  known  as 
schedule  compression  and  technology- schedule  compression,  respectively.) 

Such  schedule  normalization  techniques  have  never  been  developed.  Yet,  trying 
to  develop  estimating  relationships  without  such  normalized  schedule  data  is 
equivalent  to  building  CERs  with  then-year  cost  data. 

Once  normal  schedule  estimating  techniques  are  available,  an  FSD  cost 
estimate  can  be  allocated  to  the  schedule.  Techniques  exist  but  have  not 
been  formalized,  standardized,  or  criticized. 

Next,  the  cost  impacts  of  schedule  changes  due  to  acceleration  or  stretch¬ 
out  can  be  estimated.  Stochastic  networks  will  do  the  job.  However,  no 
technique  exists  to  estimate  the  cost  impacts  of  technology  compression.  The 
only  way  to  alert  the  reviewer  to  the  problem  is  through  increased  risk. 

This  will  be  research  area  VIII. 


CHART  4 
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Note:  Roman  numerals  in  circles  (e.g.,  (v)  )  refer  to  the 
New  Research  Areas  described  in  section  3* 


18 


3 


NEW  RESEARCH  AREAS 

From  the  framework  analysis,  nine  new  research  areas  were  defined. 
These  are  shown  on  Chart  5.  The  Roman  numerals  associated  with  each 
area  have  been  referenced  on  the  four  charts  in  the  previous  section. 

Research  Area  VII  will  serve  as  an  example  of  the  material  presented 
for  each  topic. 


CHART  5 

NEW  RESEARCH  AREAS 
ITEM 

I.  Quantify  State-of-the-Art 

II.  Extrapolation  Estimators  for  Evolutionary  Advance 

III.  Technology  Breakthrough  Evaluation  Prior  to 
DSARC  II  (Uses  V  and  VI) 

IV.  DSARC  I  Cost  Estimating  Method(s) 

V.  Technology  Impact  on  Acquisition  Cost 

VI.  Technology  Impact  on  SOA  System  Characteristics 

VII.  Normal  FSD  Schedule/Cost  Estimate  (Schedule 
Normalization) 

VIII.  FSD  Cost  Consequences  of  Technology  (Schedule) 
Compression 

IX.  Methods  to  Estimate  Advanced  Development  Costs 
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RESEARCH  AREA  VII 


"NORMAL"FSD  SCHEDULE/ COST  ESTIMATE 


DESCRI PTION :  Develop  schedule  normalization  technique(s)  which  will 
adjust  data  for  Schedule  Compression  (tight  IOC)  and  Schedule  Expansion 
(funding  cut).  Develop  "normal"  schedule  estimating  relationships. 

SIGNIFICANCE:  The  impact  of  schedule  on  cost  is  significant.  Schedule 
compressions  and  expansions  both  increase  cost.  Network  techniques 
for  estimating  these  cost  impacts  exist  once  a  "normal"  schedule  has 
been  laid  out.  However,  no  technique  exists  for  laying  out  a  "normal" 
schedule,  i.e.,  one  that  is  balanced  in  schedule  and  technical  risk 
while  minimizing  cost. 

PRODUCT  DIVISION  INTEREST  (APPLICABILITY) : 

AD  ASD  BMO  ESD  SD 

X  X  X  X  X 

PI SCUSSION :  Schedule  estimation  is  still  a  black  art.  Initial  schedules 
are  laid  out  by  the  contractor  and  then  adjusted  for  political  realities. 

No  techniques  exist  for  laying  out  a  "normal"  schedule  against  which 
contractor  and  PO  schedules  can  be  assessed.  Yet,  FSD  cost  estimates 
cannot  be  evaluated  unless  the  relationship  between  the  proposed  schedule 
and  a  "normal"  schedule  is  understood. 

Every  Product  Division  expressed  interest  in  the  schedule  and 
cost  impacts  of  schedule  changes.  Mr  Heydinger  (SD)  suggested  collecting 
data  on  schedule  growth.  He  also  wanted  to  find  ways  of  schedule  shortening 
(an  SDI  requirement) .  Mr  Townsend  (SD)  wanted  to  know  the  cost  consequences 
of  schedule  change.  Mr  Malik  (AD)  wanted  an  initial  schedule  estimate, 
sensitive  to  technology.  Mr  Scherrer  (ASD)  wanted  schedule  estimating 
techniques.  Mr  Kielpinski  (ESD)  was  concerned  with  estimating  the 
cost  of  management  reserve  as  a  function  of  schedule  compression. 

Mr  Ritchey  (ASD)  noted  that  schedule  ddata  is  being  collected  on  airframe 
production.  Development  data  will  be  collected  next.  Mr  Hansen  (BMO) 
mentioned  the  imprtance  of  schedule  normalization. 

The  first  priority  is  to  collect  consistent  schedule  data.  This 
is  being  done  at  each  of  the  Product  Divisions.  However,  a  big  technical 
problem  remains.  That  is  the  development  of  schedule  normalization 
techniques.  Without  these,  trying  to  build  schedule  estimating  relationships 
will  be  equivalent  to  trying  to  make  CERs  on  they-year  cost  data. 

Schedule  must  be  normalized  for  three  effects:  (1)  Crash  Program  (compressed 
schedule),  (2)  Accelerated  Technology  Development  (technology  compression), 
and  (3)  Budget  Cuts  (schedule  expansion). 


Once  developed,  these  normalization  techniques  ought  to  be  applied 
to  a  schedule  data  base,  thus  normalizing  the  data.  Derivation  of 
s  hedule  estimating  relationships  can  then  be  attempted. 

RELATED  AREAS:  Technology  compression  techniques  (research  area  VIII) 
call  for  the  development  of  techniques  to  adjust  normal  schedules/ cost 
when  a  technology  enhancement  is  required  to  meet  production  earlier 
than  the  normal  evoluationary  rate  of  technology  gorwth  would  permit. 
These  may  be  useful  in  schedule  normalization.  Imbedded  schedule  in¬ 
efficiencies,  other  research  area  o2,  may  have  some  overlap,  although 
this  looks  for  ways  to  reduce  schedules  rather  than  normalize  them. 

ECO  cost  estimates  and  cost/ schedule  technical  risk  costs  (#7  in  04) 
should  be  related  to  normal  schedule  or  lack  thereof.  Cost  penalty 
for  concurrency,  mentioned  in  (S4) ,  also  should  be  considered  here. 

DOABILITY : •  Uncertain.  Schedule  normalization  is  a  difficult  area 
which  needs  some  basic  research.  Normal  CER  development  procedures 
can  be  used  to  make  schedule  estimating  relationships  once  the  schedule 
data  is  normalized. 

PRIORITY: 

EXTREME  HIGH  MEDIUM  LOW 

X 

RECOMMENDATI ON :  Initiate  a  modest  effort  into  identifying  ways  to 
normalize  schedule  data.  Attempt  to  collect  schedule  data  including 
information  for  normalization. 

RESEARCH  STEPS: 

1.  Define  candidate  normali^tion  procedures. 

2.  Design  data  collection  instrument  incorporating  (1)  above. 

3.  See  if  any  existing  data  bases  contain  the  data. 

4.  If  yes,  attempt  to  build  schedule  estimating  relationships 
for  the  data. 


Sections  4-7 


Interactions  among  the  New  Research  Areas  are  presented  in  Section 
4.  Flow  charts  show  how  tools  from  the  Research  Areas  will  be  used 
in  making  the  independent  cost  estimates  and  special  cost  studies  required 
by  the  Air  Force. 

Technology  Areas  are  presented  in  Section  5.  These  include  five 
specific  technologies  plus  a  call  for  technology  roadmaps.  The  full 
discussion  of  technology  roadmaps  is  included  because  of  their  importance. 

Other  Research  Areas  are  presented  in  Section  6,  followed  by  Survey 
Areas  in  Section  7.  Evaluation  of  Estimates  made  by  PRICE  and  correcting 
for  Biased  Cost  Drivers  are  examples. 


TECHNOLOGY  AREA  T6 


TECHNOLOGY  ROADMAPS 

DESCRI PTION :  Develop  technology  and  system-oriented  roadmaps. 

SIGNIFICANCE :  Realistic  technology  projections  are  extremely  important. 

Failure  to  realize  optimistic  projections  by  contractors  results  in 
loss  of  performance  and  increased  cost. 

PRODUCT  DIVISION  INTEREST  (APPLICABILITY): 

AD  ASD  BMO  ESD  SD 

X  X  X  X  X 

PI SCUSSION :  The  detailed  work  on  system  requirements  and  technolgoy 
roadmpas  begun  at  SD  is  extremely  important.  Similiar  efforts  are 
required  at  all  Product  Divisions.  Ms  Coakley  (ESD)  recognized  the 
requirement  but  said  that  no  effort  has  been  made  by  ACC.  Technology 
forecasting  is  done  by  the  two-letter  divisions.  No  combined  list 
has  been  made. 

Mr  Scherrer  (ASD)  felt  better  technology  forecasts  from  engineering 
are  requried  so  that  good  time  estimates  can  be  associated  with  the 
new  technologies.  These  technology  time  lines  should  be  presented 
in  terms  of  the  performance  level  acheived  over  time.  Program  offices 
will  then  be  less  likely  to  prematurely  depend  on  performance  levels 
which  cannot  be  achieved  in  time.  These  timelines  will  also  set  priorities 
to  the  cost  community  in  developing  appropriate  cost  estimating  techniques. 

It  is  the  author's  opinion  that  ACCs  at  the  Product  Divisions 
should  work  with  technologists  to  translate  forecasts  into  a  usable 
roadmap  for  the  Product  Divisions.  ACC  has  the  proper  overview,  while 
technologists  tend  to  provide  too  much  detail  in  a  non-usable  form. 

RELATED  AREAS:  The  output  of  these  roadmaps  will  be  used  to  establish 
SOA  measures  (research  area  I)  and  changes  to  these  for  new  technology 
(research  area  VI).  They  will  also  be  used  to  identify  the  need  for 
schedule  (technical)  compression  (research  area  VIII). 

DOABILITY :  Space  Division  (under  AFSC  sponsorship)  has  completed  a 

microelectronics  roadmap  and  a  space  systems  roadmap.  These  need  to  be 
married  together  to  see  if  needs  and  timing  can  be  met  by  the  new  technologies. 
Other  Product  Divisions  need  to  develop  similiar  roadmaps. 

PRIORITY: 

EXTREME  HIGH  MEDIUM  LOW 

X 


23 


RECOMMENDATION :  Marry  two  SD  roadmaps  together  and  publish  as  an  example 
to  other  Product  Divisions. 

RESEARCH  AREAS; 

1.  Marry  two  roadmaps. 

2.  Write  a  tutorial  on  how  to  prepare  these  roadmaps  and 
distribute  to  other  Product  Divisions. 

3.  Describe  system,  subsystem,  and  component  SOA  for  selected 
applications  to  determine  use  in  research  areas  I,  VI, 
and  VIII. 


OTHER  RESEARCH  AREAS 


Comments  from  the  interviews  were  reviewed  and  cross-referenced 
into  topic. areas.  These  topic  areas  were  then  critically  reviewed, 
looking  for  significant  research  problems  which  had  an  expressed  interest 
from  several  Product  Divisions.  Results  of  this  review  are  contained 
in  this  section,  organized  around  the  following  topies: 

01.  Software  Size  Estimating 

02.  Historical  Schedule  Inefficiencies 

03.  Estimate  Quality 

04.  Non-Primary  Mission  Equipment  Costs 

05.  Host  Evhicle/Subsystem  Integration  Costs 

06.  Commonality  Problems 

07.  Software  Cost  Estimating 

08.  Evaluation  of  Estimates  made  by  PRICE 

In  some  instances,  the  topic  already  has  had  significant  work. 

A  critical  survey  and  distribution  of  the  results  seemed  more  appropriate 
than  a  research  task.  These  topics  are  discussed  in  section  6. 


RESEARCH  AREA  08 


EVALUATION  OF  ESTIMATES  MADE  BY  PRICE 


DESCRIPTION:  How  good  is  PRICE  for  estimating  the  costs  of  future 
advanced  systems? 

SIGNIFICANCE :  PRICE  computer  programs  have  been  used  as  a  primary 
estimating  technique  by  a  number  of  the  Product  Divisions.  The  accuracy 
of  PRICE  cannot  be  theoretically  calculated  due  to  the  balck  box  nature 
of  the  model.  Thus,  the  quality  of  many  Air  Force  cost  estimates  using 
PRICE  is  suspect. 

PRODUCT  DIVISION  INTEREST  (APPLICABILITY): 

AD  ASD  BMO  ESD  SD 

XX  XX 

DISCUSSION:  PRICE  cost  estimating  models  (both  hardware  and  software) 
have  had  increasing  use  by  the  Air  Force.  The  models  are  fairly  easy 
to  use  and  they  produce  answers  quickly  and  efficiently.  However, 
the  estimating  techniques,  scaling  rules,  etc.  which  consistitute  the 
model  are  not  visible  to  the  user.  All  that  is  known  is  that  the  model 
can  be  calibrated  to  the  user's  data  and  that  accurate  estimaes  will 
be  made  if  the  right  inputs  are  given.  Since  some  of  these  inputs 
are  judgmental  (complexity  factors),  correct  use  of  the  model  can  boil 
down  to  having  the  right  judgment. 

These  complexity  factors  are  so  important  that  users  ha”;  begun 
studying  their  nature.  PRICE  is  run  backwards,  entering  th*  ost  as 
an  input  variable,  and  calculating  the  implied  complexity  t-or.  . 

These  factors  are  called  ECIRP  factors  (PRICE  spelled  backwe  is) . 

At  AD,  PRICE  is  used  extensively  and  the  calculation  c.  ECIRP  cav_  ors 
is  part  of  any  data  collection  effort  (Mr  Soler) . 

Various  analysts  are  becoming  concerned  and  would  like  an  objective 
way  to  evaluate  PRICE.  Mr  Malik  (AD)  wants  to  know  how  good  pRICE 
is  in  forecasting  the  cost  of  technically  advanced  systems.  Mr  Back 
(ASD),  Mr  Foster  (AD),  and  Mr  Scherrer  (ASD)  all  feel  that  alternatives 
are  needed,  or  at  least  that  the  situations  for  valid  PRICE  use  need 
to  be  identified.  Capt  Dean  (ESD)  wants  to  know  how  PRICE  estimates 
software-hardware  integration  costs. 

Since  PRICE  is  calibrated,  to  date  is  seems  likely  that  it  can 
produce  good  estimates  within  the  range  of  the  data  (interpolation). 

But  what  information  does  PRICE  use  to  extrapolate?  What  are  the  scaling 
rules? 


Mr  Koscielski  (SD)  reported  that  people  are  studying  ECIRP  factors 
and  building  relationships  which  estimate  ECIRP  values  based  on  system 
physical  and  performance  characteristics.  Thus,  physical  and  performance 
characteristics  of  a  new  system  can  be  used  to  estimate  ECIRP  values. 

These  ECIRP  values  are  then  input  to  PRICE  and  the  cost  estimate  obtained. 

This  scheme  could  be  evaluated  with  techniques  developed  in  new 
research  area  II. 

RELATED  AREAS: 

Techniques  in  research  area  II  can  be  used  to  test  PRICE  extrapolation. 
DOABILITY;  Not  difficult  or  costly  if  ECIRP  data  base  exists. 

PRIORITY: 

EXTREME  HIGH  MEDIUM  LOW 

X 

RECOMMENDATION :  Conduct  an  evaluation  of  PRICE  extrapolation  capability 
using  techniques  for  research  area  II. 

RESEARCH  STEPS: 

1.  Obtain  ECRIP  data  base. 

2.  Define  area  II  evaluation  technique. 

3.  Perform  evaluation. 


SURVEY  AREAS 


Comments  from  the  interviews  were  reviewed  and  cross-referenced 
into  topic  areas.  Each  topic  area  was  then  analyzed  to  find  problems 
of  common  interest.  In  several  cases  these  problems  of  common  interest 
were  in  areas  where  significant  prior  research  has  beendone.  These 
topies  are  discussed  in  the  following  pages  with,  in  some  cases,  a 
recommendation  for  a  critical  survey  and  distribution  of  the  resutls 
to  the  Product  Divisions. 

The  following  topics  are  presented: 

51.  Correcting  for  Biased  Cost  Drivers 

52.  Impbedded  cost  Estimating  Practices 

53.  Competition  Impacts 

54.  Other  Contracting  Method  Problems 

55.  Production  Rate  Effects 

56 .  Contractor  Integration  Cost  Estimation 


SURVEY  AREA  SI 


CORRECTING  FOR  BIASED  COST  DRIVERS 


DESCRIPTION;  Conduct  a  review  of  AFSC  work  done  in  this  area.  Critique, 
document,  and  distribute  findings. 

SIGNIFICANCE ;  Cost  estimates  will  be  low  when  they  are  made  using 
models  with  cost  drivers  that  are  underestimated.  Significance  depends 
on  the  amount  of  bias  in  each  cost  driver  estimate.  Zero  to  70  percent 
is  very  likely. 

PRODUCT  DIVISION  INTEREST  (APPLICABILITY); 

AD  ASD  BMO  ESD  SD 

X  X  X  X  X 

DISCUSSION;  There  was  great  consensus  on  the  importance  of  making 
estimates  with  cost  drivers  (independent  variables)  which  are  available 
early  (Mr  Ritchey-ASD  and  Mr  Townsend-SD) .  A  corollary  is  that  the 
cost  driver  informationa  must  be  accurate  early  on.  However,  many 
cost  drivers  are  historically  estimated  low  early  in  the  system  life 
cycle.  Examples  are  number  of  test  flights  (Mr  ThompsonOESD) ,  spacecraft 
wegith  (Mr  Heydinger-SD) ,  and  number  of  circuit  boards  (Mr  Soler-AD) . 
Another  example  is  software  size  (research  area  01).  These  biased 
cost  drivers  can  be  found  by  tracking  their  estimated  value  between 
DSARC  I  and  DSARC  II  (Mr  Soler) . 

Having  identified  the  biased  cost  drivers,  how  does  one  correct 
for  them?  Three  techniques  are  possible; 

1.  Develop  unbiased  estimates  of  the  cost  drivers.  This 
is  being  suggested  for  software  size  in  research  are 
01. 

2.  Develop  correction  factors  for  the  cost  driver.  SD  (Mr 
Heydinger)  has  correction  factors  (percentage  increase) 
for  satellite  weight  at  key  milestones.  These  are  applied 
unless  the  PO  can  make  a  strong  case  that  the  historical 
cost  driver  underestimate  has  already  been  corrected 

for . 

3.  Develop  an  alternative  unbiased  cost  driver  which  is 
also  available  early  on.  Size  of  specifications  is  an 
example  for  software.  This  is  suggested  as  other  research 
area  07. 

RELATED  AREAS;  Sofware  sizing  estimate,  other  research  area  01;  and 
software  cost  model  baesd  on  specifications,  other  research  area  07; 
are  examples  of  1  and  3  above. 


DOABILITY ;  Should  be  easy  to  collect,  critique,  and  disseminate  this 
information. 

PRIORITY; 

EXTREME  HIGH  MEDIUM  LOW 

X 

RECOMMENDATION ;  A  modest  project  to  collect,  critique,  and  distribute 
information  about  this  topic  should  be  accomplished  this  next  fiscal 

year. 

RESEARCH  STEPS; 

1.  Identify  other  biased  cost  drivers  and  correction  techniques. 

2.  Review  and  critique  supportive  material. 

3.  Prepare  report. 

4.  Disseminate  to  Product  Divisions. 
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RECOMMENDATIONS 


OVERVIEW 


This  study  approached  the  question  of  new  research  areas  for  cost 
analysis  from  two  directions. 

1.  First,  the  focus  was  on  identifying  areas  where  the  cost 
community  tools  are  not  adequate.  A  framework  was  developed 
to  examine  the  quality  of  cost  analysis  with  respect 

to  both  technology  and  systems  life-cycle  phase.  Nine 
new  research  areas  were  identified  and  examined  in  section 
3.  Priorities  were  assigned  and  are  summarized  in  table 
8.1.  Although  this  assessment  was  primarily  accomplished 
by  the  author,  the  new  research  areas  were  discussed 
with  some  interviewees,  which  resulted  in  formulating 
new  research  areas  IV  and  IX  and  fine-tuning  the  other 
areas. 

2.  Second,  areas  of  major  concern  to  cost  analysts  of  the 
various  Product  Divisions  were  identified  during  the 
interviews.  Notes  from  the  interviews  were  examined 

for  consensus.  Ideas  were  grouped  into  technology  areas 
(section  5)  and  additional  research  areas.  These  additional 
research  areas  were  then  divided  into  other  research 
areas  (section  6)  and  survey  areas  (sction  7).  Survey 
areas  were  those  with  significant  prior  research  where 
a  critical  survey  should  not  only  disseminate  the  ideas 
to  the  various  Product  Divisions,  but  also  provide  proper 
review  for  identifying  the  next  research  step,  if  any. 
consensus  of  interest,  research  (survey)  potential,  and 
doability.  This  resulted  in  a  priority  raking  shown 
in  table  8.1. 

The  interview  process  also  surfaced  a  great  deal  of  interest  in 
topics  that  were  not  suitable  for  cost  research  and  were  therefore  excluded. 
The  following  are  examples: 

Data  collection  was  a  concern  of  ten  interviewees.  Topics  varied 
but  the  need  for  more  data  was,  as  always,  a  principal  concern.  Furthermore, 
it  was  asserted  that  data  should  be  maintained,  not  just  collected. 

Data  collection  standardization  was  a  concern  for  nine  interviewees. 

Data  collection  alone  is  not  sufficient.  Standard  WBSs  across  systems 
are  required.  Cost  models  and  reports  should  use  the  same  standardized 
format.  Then  analyses  and  comparisons  can  be  accomplished. 
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Source-selection  support  was  another  concern.  The  cost  analyst  can 
provide  invaluable  support  in  normalization  of  bids  and  identifying  bid 
rates  with  significant  bottom  line  cost  impacts.  The  cost  analyst  often 
sees  the  forest  while  negotiators  are  examining  the  trees.  Another 
service  that  the  cost  analyst  can  provide  is  to  estimate  the  impacts  of 
award  selections  upon  burden  rates  for  other  contractor  in-house  DOD 
work . 

Also  excluded  from  this  review  were  (1)  O&S  costs,  (2)  institutional 
and  procedural  changes,  (3)  automation  to  increase  estimating  efficiency, 

(4)  technical  recruiting,  and  (5)  Black  world  applications.  These  were 
excluded  from  the  outset  and  were  therefore  not  discussed  in  the  intereviews 

SPECIFIC  RECOMMENDATIONS 

The  priority  rankings  in  table  8.1  give  an  order  to  the  pursuit  of 
these  research  areas.  The  extreme  priority  calls  for  study  immediately. 

A  high  priority  recommends  studies  during  the  fiscal  year.  Medium  are 
those  areas  which  should  be  explored,  funding  permitting.  Low  indicates 
an  area  that  does  not  have  enough  common  interest  or  application  to  be 
pursued  at  the  AFSC  level.  A  low  rating  does  not  assert  that  the  idea 
was  not  worth  pursuing,  but  rather  that  the  investigation  should  be 
pursued  by  a  more  focused  user  group. 

Rankings  within  each  of  the  priority  classes  were  then  made  and 
reasons  for  the  ranking  given.  These  are  presented  in  the  full  text, 
available  in  early  October. 


Table  8.1 

SUMMARY  OF  RESEARCH  AREA  PRIORITIES 


Extreme 

High 

Medium 

Low 

Research  Areas 

I. 

Quantify  State  of  the  Art 

X 

II. 

Extrapolation  Estimators  for 
Evolutionary  Advance 

X 

III. 

Technology  Breakthrough 

Evaluation  Prior  to  DSARC  II 

X 

IV. 

DSARC  I  Cost 

Estimating  Methods 

X 

V. 

Technology  Impact  on 

Acquisition  Costs 

X 

VI. 

Technology  Impact  on  SOA 

System  Characteristics 

X 

VII. 

Normal  FSD  Schedule/Cost 

Estimate 

X 

VIII. 

FSD  Cost  Consequences  of 

Technology( Schedule)  Compression 

X 

IX. 

Methods  to  Estimate  Advanced 
Development  Cost 

X 

Technology  Areas 

Tl. 

VHSIC 

X 

T2 . 

CAD 

X 

T3. 

CAM 

X 

T4. 

GaAs 

X 

T5. 

Composite  Materials 

X 

T6. 

Technology  Road  Maps 

X 

Table  8.1  (Continued) 


Extreme 

High 

Medium 

Low 

Other  Research  Areas 

01. 

Software  Size  Estimating 

X 

02. 

Historical  Schedule  Inefficiencies 

X 

03. 

Estimate  Quality 

X 

04. 

SEPM  Model 

X 

03. 

Platform  Integration  Costs 

X 

06. 

Commonality  Problems 

X 

07. 

Software  Costs  as  Function  of 

Software  Functional  Specification 

X 

08. 

Evaluate  PRICE/ECIRP  for 

Technically  Advanced  Systems 

X 

Survey 

Areas 

SI. 

Correcting  for  Biased  Cost  Drivers 

X 

S2. 

Imbedded  Cost  Estimating  Practices 

X 

S3. 

Competition  Impacts 

X 

S4 . 

Multiyear  Procurement  and  Warranties 

X 

S5 . 

Production  Rate  Effects 

X 

S6. 

Contractor  Integration  and 

Checkout  Cost  at  All  Levels 

X 

SUMMARY 


The  Road  Map  is  an  important  document  in  helping  to  prioritize  funding 
of  CMIP  projects.  It  will  allow  AFSC  to  better  spend  its  research 
funds  and  project  where  research  is  needed  to  improve  it's  cost  estimating 
"tool  box".  The  bottom  line  is  to  ensure  that  AFSC  continues  to  deliver 
quality  cost  estimates  for  acquiring  new  weapon  systems.  As  technology 
changes  so  must  the  methods  of  estimating  the  systems/ technology. 

If  the  Road  Map  is  to  continue  to  be  of  use  it  must  be  regularly  updated. 
Once  the  Road  Map  is  used  to  prioritize  a  given  year's  projects  and 
budgets,  the  results  of  ongoing  projects,  changes  in  technology  and 
the  way  systems  are  acquired  must  be  reevaluated  to  reflect  the  needs 
for  the  following  years  set  of  projects.  The  Road  Map  is  a  living 
document  of  the  cost  estimating  needs  of  the  Air  Force. 
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